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(54) Control system and method for controlling object using emotions and personality generated 
in the object 

(57) An autonomous device behaving adaptively to 
a user, comprising: a recognition unit which recognizes 
a user-related status; a personality-forming unit which 
establishes personality composed of preselected per- 
sonality elements based on the user-related status 
accumulated for a predetermined period of time based 
on a predetermined relationship between the selected 
personality elements and the user-related status; an 
emotion-forming unit which establishes emotions com- 
posed of preselected emotion elements based on the 
user-related status and the personality based on a pre- 
determined relationship between the preselected emo- 
tion elements, the user-related status, and the 
personality; and an autonomous behavior-establishing 
unit which selects a pattern of autonomous behavior 
based on the user-related status, the personality, and 
the emotions, based on a predetermined relationship 
between patterns of autonomous behavior, the user- 
related status, the personality, and the emotions. 
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Description 

[0001] This invention relates to a system for control- 
ling an object interacting with a user and environment, 
and particularly to that for adapting, to a user, behavior s 
of an object capable of behaving in an autonomous 
manner, thereby creating behavior highly responsive to 
the user or environment. 

[0002] Heretofore, various controlling methods have 
been available for controlling an object in accordance w 
with a user's demands. 

[0003] In such controlling methods, normally, the user 
sets a desired target value at output, and the object is 
controlled in such a way that the output matches the tar- 
get value, while feeding the output back to a control sys- is 
tern which compares the feedback and the target value 
to adjust the output, in the above, by feeding the output 
back to the system to adjust the output, the output of the 
object to be controlled can approach the target value, 
thereby achieving control satisfying the user's prefer- 20 
ences. 

[0004] If a target value is inputted by the user, it is rel- 
atively easy to obtain output close to the user's prefer- 
ences. For example, a temperature of air controlled by 
an air conditioner is an easy object because the rela- 25 
tionship between output (cooling or heating air) and a 
target value (temperature) is simple, and thus anyone 
can set a thermostat of the air conditioner at a target 
value. However, if the relationship between output and a 
target value is complicated, especially in a control sys- 30 
tern wherein changing a target value affects not only 
one output but also other outputs, only a skilled person 
can change the target value. 

[0005] Further, since humans have emotions and 
such emotions affect momentary preferences, for exam- 35 
pie, even if an object or device is designed specifically 
for one individual user, it would be difficult to control the 
object in such a way as to always satisfy the user. 
[0006] In the above conventional control system, out- 
put from the object is designed to always approach the 40 
target value. Thus, the longer the object is used, the 
easier the prediction of behavioral patterns becomes for 
the user, if the user can predict the output from the 
object, the object cannot be, used for creative assist- 
ance to the user's intellectual work or desire. 45 
[0007] An objective of the present invention is to solve 
the above problems associated with conventional con- 
trol systems, and to provide a control system which 
allows outputting an adequate value ultimately giving 
the user more satisfaction than does a value obtained so 
from the user's direct order, particularly using emotions 
or pseudo-emotions and a personality or pseudo-per- 
sonality generated in the object in response to the user 
and environment. The object can develop with respect 
to individuality in accordance with the user or with use 55 
by the user. 

[0008] One important aspect of the present invention 
attaining the above objective is to provide to an autono- 



mous device behaving adaptively to a user, comprising: 
a recognition unit which recognizes a user-related sta- 
tus; a personality-forming unit which establishes per- 
sonality composed of preselected personality elements 
based on the user-related status accumulated for a pre- 
determined period of time based on a predetermined 
relationship between the selected personality elements 
and the user-related status; an emotion-forming unit 
which establishes emotions composed of preselected 
emotion elements based on the user-related status and 
the personality based on a predetermined relationship 
between the preselected emotion elements, the user- 
related status, and the personality; and an autonomous 
behavior-establishing unit which selects a pattern of 
autonomous behavior based on the user-related status, 
the personality, and the emotions, based on a predeter- 
mined relationship between patterns of autonomous 
behavior, the user-related status, the personality, and 
the emotions. 

[0009] In the above, the external information or envi- 
ronment information can be omitted if the device is used 
under limited circumstances or if the external or envi- 
ronment information has nothing to do with accomplish- 
ment of the user's command or if the user so desires. 
[0010] The device to be controlled in the present 
invention is not limited, and any device such as an 
engine in a motorbike or a car, an outboard engine in a 
vessel, an industrial robot used in machine tools, a Pet 
robot used at home as a pet, a motor used in electri- 
cally-driven vehicles, a touring assist system equipped 
in a car, a conversation robot, a research assistance 
system, or the like can be controlled by adopting the 
control system of the present invention based on the 
same principle as the above. By modifying items 
included in information to be processed, the control 
device can be controlled in the same way. In any case, 
according to the present invention, the control device, to 
which the present invention is adapted, is likely to be 
equipped with quasi-intelligence. 
[001 1 ] In the present invention, another objective is to 
provide a method for adapting behavior of a device to 
user's characteristics, to which method the aforesaid 
device is adapted. 

[0012] According to the present invention, this objec- 
tive is solved by a method for adapting behavior of a 
device to a user's characteristics, comprising the steps 
of: (b) recognizing a user-related status; (d) establishing 
personality composed of preselected personality ele- 
ments based on the user-related status accumulated for 
a predetermined period of time based on a predeter- 
mined relationship between the preselected personality 
elements and the user-related status; (f) establishing 
emotions composed of preselected emotion elements 
based on the user-related status and the personality 
based on a predetermined relation between the prese- 
lected emotion elements, the user-related status, and 
the personality; and (g) selecting a pattern of autono- 
mous behavior based on the user-related status, the 
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personality, and the emotions based on a predeter- 
mined relationship between patterns of autonomous 
behavior, the user-related status, the personality, and 
the emotions. 

[0013] Other preferred embodiments of the present 
invention are laid down in further dependent claims. 
[0014] In the following, the present invention is 
explained in greater detail with respect to several 
embodiments thereof in conjunction with the accompa- 
nying drawings, wherein: 

Figure 1 is a schematic block diagram illustrating a 
functional architecture of the control system 
according to the present invention; 

Figure 2 is a schematic block diagram showing an 
example of the entire system of an assist system 
according to the present invention; 

Figure 3 is a schematic flow chart showing the 
entire system of the assist system according to the 
present invention; 

Figure 4 Is a schematic block diagram showing the 
user/external information recognition unit A indi- 
cated in Figure 1; 

Figure 5 is a schematic flow chart showing proc- 
esses at the pseudo-personality information-setting 
unit B indicated in Figure 1 ; 

Figure 6 is a schematic flow chart showing proc- 
esses at the pseudo-emotion generation unit C indi- 
cated in Figure 1; 

Figure 7 is a schematic flow chart showing proc- 
esses at the emotion expression decision unit D 
indicated in Figure 1 ; 

Figure 8 is a schematic flow chart showing proc- 
esses at the autonomous behavior generation unit 
E indicated in Figure 1 ; 

Figure 9 is a schematic flow chart showing proc- 
esses at the behavior planning unit G indicated in 
Figure 1 ; 

Figure 10 is a schematic flow chart showing proc- 
esses from generation of each item of behavior 
information based on the current behavior informa- 
tion to setting of final output behavior; 

Figure 11 is a schematic block diagram showing an 
example of the entire system of a pet robot accord- 
ing to the present invention; 

Figure 12 is a schematic flow chart showing the 
entire system of the pet robot according to the 



present Invention; 

Figure 13 is a schematic block diagram showing the 
user/environment information recognition unit A 
s indicated in Figure 1 1 ; 

Figure 14 is a schematic flow chart showing proc- 
esses at the pseudo-personality information-setting 
unit B indicated in Figure 1 1 ; 

10 

Figure 15 is a schematic flow chart showing proc- 
esses at the pseudo-emotion generation unit C indi- 
cated in Figure 1 1 ; 

15 Figure 16 is a schematic flow chart showing proc- 
esses at the emotion expression decision unit D 
indicated in Figure 1 1 ; 

Figure 17 is a schematic flow chart showing proc- 
20 esses at the autonomous behavior generation unit 
E indicated in Figure 1 1 ; 

Figure 18 is a schematic flow chart showing proc- 
esses at the behavior planning unit G indicated in 
25 Figure 1 1 ; and 

Figure 19 is a schematic flow chart showing proc- 
esses from generation of each item of behavior 
information based on the current behavior informa- 
30 tion to setting of final output behavior. 

[0015] The device according to the present invention 
is an intelligent, adaptive, user-friendly device. The term 
"intelligent" means that the device is able to perceive its 

35 surroundings, reason out its task, plan a course of 
action, and execute those actions. The term "adaptive" 
means both that the device is able to respond to unex- 
pected situations while it Is carrying out its plans and 
also that it is able to learn from its experiences and 

40 improve its performance on the tasks that it commonly 
carries out. The term "user-friendly" means not only that 
the user is able to interact in a natural manner with the 
device but also that the device comes to be viewed 
more as a "friend" or a "pet" than a "machine". In order 

45 to accomplish these objectives, the device has a 
pseudo-personality and pseudo-emotions, interacts 
with its user on an affective level, and learns about its 
user, in addition to autonomously and spontaneously 
pursuing its own goals. 

50 

Basic Architecture 

[0016] The control system for controlling an object 
using pseudo-emotions and a pseudo-personality of the 
55 present invention will be explained with reference to 
several embodiments depicted in the figures. Figure 1 is 
a schematic block diagram illustrating a functional archi- 
tecture of the control system according to the present 
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invention. 

[001 7] In this figure, an autonomous device behaving 
adaptively to a user, comprises: a sensing unit which 
senses signals of user's command and predetermined 
user-related signals relevant to accomplishment of the 
command but not part of the command; a recognition 
unit which recognizes the user's command and a cur- 
rent user-related status based on the sensed signals of 
the command and the sensed predetermined signals; a 
first memory which stores and updates the recognized 
command and the recognized user-related status; a 
pseudo-personality-forming unit which establishes each 
element of predetermined pseudo-personality elements 
based on the user-related status accumulated for a pre- 
determined period of time in the first memory to form 
pseudo-personality, said pseudo-personality-forming 
unit memorizing a predetermined relationship between 
the predetermined pseudo-personality elements and 
the user-related status; a second memory which stores 
and updates the formed pseudo-personality, wherein 
updating takes place under predetermined conditions; a 
pseudo-emotion-forming unit which establishes each 
element of predetermined pseudo-emotion elements 
based on the user-related status stored in the first mem- 
ory and the pseudo-personality stored in the second 
memory to form pseudo-emotions, said pseudo-emo- 
tion-forming unit memorizing a predetermined relation- 
ship between the predetermined pseudo-emotion 
elements, the user-related status', and the pseudo-per- 
sonality; an autonomous behavior-establishing unit 
which selects a pattern of autonomous behavior based 
on the user-related status, the pseudo-personality, and 
the pseudo-emotions, said autonomous behavior-form- 
ing unit memorizing a predetermined relationship 
between patterns of autonomous behavior, the user- 
related status, the pseudo-personality, and the pseudo- 
emotions; a commanded behavior-establishing unit 
which constitutes a pattern of commanded behavior in 
accordance with the recognized user's command; a 
behavior control unit which controls behavior by com- 
bining the selected pattern of autonomous behavior and 
the constituted pattern of commanded behavior based 
on a predetermined priority order; and an output device 
which outputs the controlled behavior. 
[0018] In the above, preferably, user-unrelated infor- 
mation such as environmental information and external 
database information is included in input information in 
order to establish complex behavior in the object. 
[0019] In the above, preferably, the device further 
comprises a third memory, linked to the behavior control 
unit, which stores and updates the pseudo-emotion 
formed by the pseudo-emotion-forming unit, wherein 
the second memory is further linked to the behavior 
control unit, wherein the pseudo-emotion stored in the 
third memory and the pseudo-personality stored in the 
second memory are used as parameters of the prede- 
termined priority order provided in the behavior control 
unit (indicated with dotted lines in Figure 1). Further, the 



first memory may be further linked to the behavior con- 
trol unit, wherein the user-related status stored in the 
first memory, the pseudo-personality stored in the sec- 
ond memory, and the pseudo-emotion stored in the third 
5 memory are used as parameters of the predetermined 
priority order provided in the behavior control unit. In the 
above structures, the output behavior can be more 
autonomous and adaptive (indicated with a double-dot- 
ted line in Figure 1). 

10 [0020] Figures 2-10 show an embodi ment wherein the 
present invention is adapted to a touring assist system 
which assists a user when the user works out a plan of 
touring or travelling. Figures 11-19 show an embodi- 
ment wherein the present invention is adapted to a pet 

15 robot which a user can keep as a pet. However, the 
present invention is not limited to these embodiments, 
and this control system can be adapted to any device 
which is useful to the user as an intelligent, adaptive, 
user-friendly device. The point is that appropriate input 

20 and output are selected, depending on the goal of the 
device and the user's preferences. In any case, the prin- 
ciple of the control system is the same as those illus- 
trated in the figures. 

25 Touring Assist System 

[0021 ] This touring assist system (hereinafter referred 
to as "assist system") is designed to assist a user when 
the user lays out a plan of a travel or tour by motorbike, 

30 car, train, or on foot. The assist system itself has a 
pseudo-emotion which changes in response to interac- 
tion with the user or depending on the surrounding envi- 
ronments, and a pseudo-personality which grows based 
on the interaction with the user or its experience. 

35 Accordingly, the assist system assists working out the 
user's plan based on the interaction with the user and 
information on the environments (train maps, road 
maps, traffic news, etc., while the device expresses its 
pseudo-emotion or produces an autonomous response 

40 or a response in accordance with the user's command. 
[0022] In practice, in order to recognize the state of 
the user or the environments, this assist system com- 
prises a visual detection sensor, a touch detection sen- 
sor, an auditory detection sensor, and a means for 

45 accessing external databases, and further comprises 
an output device, such as a visual output device and an 
auditory output device, for indicating its pseudo-emo- 
tions or responses to the user. The assist system can be 
constructed so as to load a motorbike or a car therewith 

so or allow the user to carry it. 

Basic Structures of Touring Assist System 

[0023] Figure 2 is a schematic block diagram showing 
55 an example of the entire system of an assist system 
according to the present invention. As shown in this fig- 
ure, the assist system comprises: a user/external infor- 
mation recognition unit A for recognizing a state of the 
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user or external information; a pseudo-personality infor- 
mation-setting unit B for setting a pseudo-personality by 
referring to the outcome of recognition by the recogni- 
tion unit A; a pseudo-emotion generation unit C for gen- 
erating pseudo-emotions based on the outcome of 
recognition by the recognition unit A and the information 
on the pseudo-personality set by the pseudo-personal- 
ity information-setting unit B; an emotion expression 
decision unit D for deciding manners of emotion expres- 
sion, such as expression methods to express laugh or 
anger on a display, based on the information on the 
pseudo-emotion: an autonomous behavior generation 
unit E for generating autonomous behavior based on 
the outcome of recognition, the information on the 
pseudo-personality, and the information on the pseudo- 
emotions; a commanded behavior generation unit F for 
generating behavior in accordance with the user's com- 
mand based on the outcome of recognition and at least 
the preceding behavior previously outputted by this sys- 
tem itself, and a behavior planning unit G for deciding 
final behavior to be outputted using output from each of 
the emotion expression decision unit D, the autono- 
mous behavior generation unit E, and the commanded 
behavior generated unit F, based on the outcome of rec- 
ognition, the information on the pseudo-personality, and 
the information on the pseudo-emotions. As shown in 
Figure 3, once the system is activated, this assist sys- 
tem repeats each processing step at units A through G 
until the system is turned off. The structure and function 
of each unit will be explained in detail below. 

User/External Information Recognition Unit 

[0024] As shown in Figure 2, the user/external infor- 
mation recognition unit A comprises various input 
devices (e.g., an auditory input device, a visual input 
device, an evaluation input device, a keyboard, a com- 
munication device, a thermometer, an illuminometer, 
and/or a watch), a user's evaluation recognition unit Al, 
a user's emotion recognition unit A2, a user's command 
recognition unit A3, an external information recognition 
unit A4, and a current user information/characteristics 
deduction unit A5. The user's evaluation recognition unit 
A1 performs recognition-processing based on the infor- 
mation obtained via the input devices. The current user 
information/characteristics deduction unit AS deter- 
mines or deduces a current state of the user and the 
user's characteristics such as the user's preferences 
based on the recognition outcome from the user-related 
recognition units A1 through A3. 
[0025] In the above, the input device includes a micro- 
phone as an auditory input device, a camera as a visual 
input device, a switch or lever for inputting the user's 
evaluation as an evaluation input device and an 
antenna, a modem, a terminal adapter, or merely an 
input port as a communication device. However, the 
input devices are not limited to the above. Any input 
device can be used which is available on the market, 



which is capable of detecting external information visu- 
ally, auditorily, and/or tactilely, and which is capable of 
accessing external databases. 

[0026] The term "external information" means not only 
5 information on the surrounding environments, such as 
the temperature, brightness, or time, where the system 
is operated (hereinafter referred to as "environmental 
information"), but also information obtained from exter- 
nal databases other than from the user or the system. 
10 [0027] Figure 4 is a schematic block diagram showing 
the user/external information recognition unit A in detail 
to explain processes at each of the processing units A1 
through A5 in the user/external information recognition 
unit A. 

15 [0028] As shown in this figure, the user's evaluation 
recognition unit A1 establishes the user's evaluation as 
to behavior outputted by the system and sets the result, 
for example, as information on the user's evaluation, by 
calculating an evaluation value from data inputted by an 

20 evaluation input device composed of a lever, for exam- 
ple, which allows the user to input his/her evaluation at 
an appropriate level among plural levels, depending on 
the user's satisfaction. 

[0029] The user's emotion recognition unit A2 pre- 

25 sumes the user's current emotions based on the char- 
acteristics found in the auditory information and the 
user's facial expression, wherein an auditory analysis 
(analysis of frequency distributions or pitches of the 
sound/voice) and a facial expression analysis (analysis 

30 based on the shapes of the eyes and the mouth) are 
conducted based on the auditory information and the 
user's facial expression inputted from the auditory input 
device and the visual input device. Further, based on 
the presumed emotions, the user's emotion recognition 

35 unit A2 determines the user's current emotion. In the 
above, the several patterns of emotions are obtained by 
categorizing in advance the emotion expressions of the 
user into several patterns of emotions such as joyful, 
sad, surprised, angry, disgusted, and fearful. The pat- 

40 terns of emotion expressions are recognized by subdi- 
viding each pattern into several levels. In the above, the 
presumption of the user's emotions based on the input 
information can be per-formed using a fuzzy logic with 
predetermined rules, or using data obtained by formu- 

45 lating or mapping in advance the relationship between 
emotions and characteristics of sounds/voices and 
facial expressions, or using appropriate functional equa- 
tions if regularity can be found in the relationship 
between emotions and characteristics of sounds/voices 

so and facial expressions. The information inputted into the 
emotion recognition unit A2 is not limited to the above 
and includes any information related to the user's 
behavior affected by or representing the user's emo- 
tions, for example, the manner of punching the key- 

55 board: If the user punches the same key several times 
without a special reason when the user inputs informa- 
tion, the user is presumed to be irritated (a low level of 
anger). 
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[0030] The user's command recognition unit A3 rec- 
ognizes and establishes the user's command informa- 
tion based on indirect command information obtained 
by analyzing the meaning of the sound/voice recog- 
nized by the auditory input device and by analyzing 5 
information on the user's gestures recognized by the 
visual input device, and based on direct command infor- 
mation inputted via the keyboard. The command infor- 
mation includes not only direct commands such as 
"search" or "lay out an action plan" but also a degree of w 
requirement for the command such as "urgent" and 
"normal". The degree of requirement can be recognized 
by, for example, the loudness of the user's voice or. the 
speed of the user's speech inputted via the auditory 
input device, or the user's gesture inputted via the visual 75 
input device. The recognition means is not limited to the 
above and includes various input information such as 
the strength of punching the keyboard wherein each key 
is provided with a pressure-sensing sensor. The infor- 
mation obtained from the keyboard provided with pres- 20 
sure-sensing sensors can be used in the recognition 
process at the user's emotion recognition unit A2. 

[0031] The external information recognition unit A4 
establishes external information such as newly released 
information from external databases, the user's past 25 
record of using the system, ambient temperature, 
brightness, time, and date, by automatically collecting 
information through external information networks such 
as the Internet via the communication device and by 
processing the inputted information from various sen- 30 
sors such as a thermometer, illuminometer, and watch. 
[0032] The current user information/characteristics 
deduction unit A5 determines the deducing the current 
state and the characteristics of the user based on the 
recognition results at the user-related information rec- 35 
ognition units A1 through A3, the information on the cur- 
rent priority behavior described later, the information on 
the emotion expression patterns, and the information on 
the autonomous behavior patterns. The current user 
information/characteristics deduction unit AS then 40 
updates data of the recognition information related to 
the user stored in a user information memory 1 . 
[0033] In the above, the current state of the user 
includes a time period while the user is located within a 
range where the system can locate the user; the level of 45 
the user's emotions; and the user's evaluation as to the 
behavior outputted by the system. The characteristics of 
the user includes frequency of the occurrence of each 
emotion, the changing rate or range of emotions, the 
types of keywords that the user uses, frequency of use so 
of keywords, and the average value of the user's evalu- 
ation as to the behavior outputted by the system. 
[0034] The output of the user's command recognition 
unit A3 is transmitted not only to the user information 
memory 1 but also to a user's command information 55 
memory 2, thereby updating the information related to 
the user's command stored in the memory 2. The infor- 
mation related to the user's command includes, for 



example, the contents of the command, the object of the 
command, and the conditions of the command. 

[0035] The output of the external information recogni- 
tion unit A4 is transmitted to an external information 
memory 3, thereby updating the information stored in 
the memory 3. 

Pseudo-Personality information-Setting Unit 

[0036] Figure 5 is a schematic flow chart showing 
processes at the pseudo-personality information-setting 
unit B. 

[0037] The pseudo-personality information-setting 
unit B is constructed so as to set, as a pseudo-person- 
ality, a group of several preliminary personalities, each 
subdividable into several levels. Based on the informa- 
tion stored in the user information memory 2, the level of 
each preliminary personality is set, and the higher the 
level, the more the preliminary personality contributes to 
them formation of the pseudo-emotions and the action 
plan. The preliminary personalities include, for example, 
"rise and fall of emotions", "obedience", "aggressive- 
ness", "curiosity", and "cheerfulness". Each preliminary 
personality is correlated with each characteristic of the 
user stored in the user information memory 1. In the 
above, "rise and fall of emotions" is correlated with the 
changing rate or range of the user's emotions, "obedi- 
ence" is correlated with frequency of the occurrence of 
the user's emotion "anger", "aggressiveness" is corre- 
lated with the evaluation of the behavior outputted by 
the system and frequency of the occurrence of each 
emotion in the user, at the time of evaluation, "curiosity" 
is correlated with the range of keywords that the user 
likes to use, and "cheerfulness" is correlated with fre- 
quency of the occurrence of the users emotion "joy" and 
positive evaluation of the behavior outputted by the sys- 
tem. Thus, in the above, if the emotions of the user 
intensively and often fluctuate, the level of rise and fall of 
emotions is increased. If the user gets angry often, the 
level of obedience is decreased, if the user positively 
evaluates the behavior outputted by the system and is 
delighted, the level of aggressiveness is increased. If 
the keyword that the user likes are scattered in various 
fields, the level of curiosity is increased. If the user often 
positively evaluates the behavior outputted by the sys- 
tem and is often delighted, the level of cheerfulness is 
increased. 

[0038] The pseudo-personality established at the 
pseudo-personality setting unit B is outputted to a 
pseudo-personality information memory 4 per prelimi- 
nary personality, thereby updating the information 
stored in the pseudo-personality information memory 4. 
Updating can be conducted by using the input from the 
pseudo-personality setting unit B as new pseudo-per- 
sonality information and overriding the preceding infor- 
mation, or by adding the levels of the preliminary 
personalities obtained at the pseudo-personality setting 
unit B to the preceding information. The former method 
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allows the pseudo-personality of the system to be highly 
responsive to the user's characteristics, and the latter 
method allows the pseudo-personality of the system to 
grow in accordance with frequency of use of the system 
and a lapse of time, thereby allowing the user to enjoy 5 
the growth of the pseudo-personality of the system. In 
particular, in the case of the latter method, if the levels 
of each preliminary personality are set at several 
detailed levels, and output from the pseudo-personality 
information-setting unit B is set to be low, it is possible w 
to enjoy the growth of the pseudo-personality of the sys- 
tem which takes place over a long period of time, i.e., 
the personality can be more human-like. 

Pseudo-Emotion Generation Unit 15 

[0039] Figure 6 is a schematic flow chart showing 
processes at the pseudo-emotion generation unit C. 
[0040] The pseudo-emotion generation unit C 
receives output from the user information memory 1 , the 20 
external information memory 3, and the pseudo-person- 
ality information memory 4, and, based on the above 
information, the pseudo-emotion generation unit C gen- 
erates pseudo-emotion information of the system. 
[0041 ] The pseudo-emotion generation unit C memo- 25 
rizes several predetermined preliminary emotions (such 
as "joyful", "sad", "surprised", "angry", "disgusted", and 
"fearful"), and sets the level of each preliminary emotion 
based on the information inputted from the user infor- 
mation memory 1 and the external information memory 30 
3. In practice, the level of each preliminary emotion can 
be determined by rules or functional equations correlat- 
ing the level of each preliminary emotion with the infor- 
mation on the state of the user (such as emotion level 
and evaluation level) stored in the user information 35 
memory 1 and the information on the environmental 
information stored in the external information memory 
3. The relationship between the preliminary emotions, 
the state of the user, and the environmental information 
is formulated, for example, as follows: The higher the 40 
level of "anger" among the user's emotions, the higher 
the level of "sadness" becomes and the lower the level 
of "joy" becomes among the user's preliminary emo- 
tions. The higher the level of evaluation as to the behav- 
ior outputted by the system, the higher the level of "joy" 45 
becomes and the lower the level of "sadness" becomes. 
The lower the ambient temperature obtained from the 
environmental information, the higher the level of "dis- 
gust" becomes. The darker the surrounding environ- 
ment, the higher the level of "fear" becomes. so 
[0042] The pseudo-emotion generation unit C decides 
whether or not each preliminary emotion generated at 
the unit C is outputted therefrom as pseudo-emotions 
based on a predetermined threshold set for each pre- 
liminary emotion (hereinafter referred to as "preliminary 55 
emotion threshold"). The level of a preliminary emotion 
which is not outputted is set at zero, and the level of a 
preliminary emotion which is outputted is set at an 



appropriate level based on the information on the 
pseudo-personality. A group of the above preliminary 
emotions are outputted as a pseudo-emotion. 

[0043] The preliminary emotion threshold is correlated 
in advance to the pseudo-personality information, and is 
decided based on the pseudo-personality information 
inputted from the pseudo-personality information mem- 
ory 4. In practice, each preliminary emotion threshold is 
correlated in such a way as to change in accordance 
with the level of each preliminary emotion in the 
pseudo-personality information. For example, the 
higher the level of "rise and fall of emotions" in the 
pseudo-personality, the lower the threshold of each pre- 
liminary emotion becomes. In contrast, the lower the 
level, the higher the threshold of each preliminary emo- 
tion becomes. The higher the level of "cheerfulness" in 
the pseudo-personality, the lower the threshold of only 
"joy" among the others and the higher the thresholds of 
the remaining preliminary emotions become. In the 
former, the higher the level of "rise and fall of emotions" 
in the pseudo-personality, the higher the possibility of 
exceeding the respective thresholds becomes, thereby 
increasing the possibility of expressing the preliminary 
emotions as the pseudo-emotion, i.e., emotion fluctua- 
tion becomes remarkable. In the latter, the higher the 
level of "cheerfulness" in the pseudo-personality, the 
higher the possibility of exceeding the threshold of only 
"joy" becomes, thereby rendering the system joyful. In 
another example, the higher the level of "obedience" in 
the pseudo-personality, the higher the threshold of each 
preliminary emotion becomes. In contrast, the lower the 
level of "obedience", the lower the threshold of each 
preliminary emotion becomes. As described above, by 
correlating the preliminary emotion thresholds with the 
levels of each preliminary emotion in the pseudo-per- 
sonality, the likelihood of appearance of each pseudo- 
emotion is changed depending on the pseudo-personal- 
ity, and as the pseudo-personality grows, the manner of 
appearance of pseudo-emotions grows. 
[0044] After calculating the threshold of each prelimi- 
nary emotion for each preliminary emotion, or during 
such calculation, each preliminary emotion threshold 
and the corresponding preliminary emotion are com- 
pared, and only preliminary emotions whose levels are 
higher than the respective preliminary emotion thresh- 
olds undergo calculation of their levels to output as a 
pseudo-emotion. The remaining preliminary emotions 
are cleared except for the set information. 
[0045] The levels to be outputted as pseudo-emo- 
tions are correlated with the levels of the preliminary 
personalities in the pseudo-personality, and can be cal- 
culated using rules or functional equations. In practice, 
for example, the higher the level of "rise and fall of emo- 
tions" in the pseudo-personality, the closer to the levels 
of preliminary emotions the output levels of pseudo- 
emotions become. In contrast, the lower the level of 
"rise and fall of emotions" in the pseudo-personality, the 
lower then the levels of preliminary emotions the output 
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levels of pseudo-emotions become. 

[0046] The pseudo-emotions obtained by the above 
processes are outputted to a pseudo-emotion informa- 
tion memory 5, thereby updating the data stored in the 
memory 5. 5 

Emotion Expression Decision Unit 

[0047] Figure 7 is a schematic flow chart showing 
processes at the emotion expression decision unit D. w 
[0048] The emotion expression decision unit D judges 
the level of each preliminary emotion from the pseudo- 
emotion information obtained from the pseudo-emotion 
memory 5, and determines emotion expression to be 
outputted based on preliminary emotions whose levels is 
are high or more than zero. 

[0049] The emotion expression is selected from plural 
patterns of emotion expression in relation to pseudo- 
emotions stored in advance in an emotion expression 
pattern memory 6. The emotion expression includes, in 20 
practice, changes in shapes of the eyes and the mouth 
or the shedding of tears if a "face" is created by a visual 
output device. Further, emotion expression can be indi- 
cated by background color of the "face" on the display, 
or by sounds or voices or conversation using an audi- 25 
tory output device. In the above, the emotion expression 
pattern memory 6 memorizes in advance patterns of 
each of eyes, mouth, tears, background color, sound, 
and conversation in relation to each level of each pre- 
liminary emotion. 30 
[0050] The method of selecting the patterns of the 
emotion expressions at the emotion expression unit D 
can be simply a method wherein the pattern of the emo- 
tion expression, which corresponds to the preliminary 
emotion having the highest level, is selected. However, 35 
preferably, a pattern of the absence of expression 
(expressionless) is created among patterns of (emotion 
expressions, and if the levels of all of the preliminary 
emotions are zero, the expressionless pattern is 
selected, and if the levels of preliminary emotions, 40 
except for one preliminary emotion, are zero, a pattern 
corresponding to the one preliminary emotion is 
selected. If the levels of more than one but not all pre- 
liminary emotions are more than zero, a pattern of a 
preliminary emotion having a higher level than those of 45 
the other preliminary emotions has priority over the 
other patterns. Further, among the remaining patterns, 
a pattern of a preliminary emotion having a lower level 
than the first one is selected where the first pattern does 
not apply. In the above, the first pattern and the second so 
pattern are combined to make one combined expres- 
sion. If the levels of all of the preliminary emotions are 
more than zero, the expressionless pattern may be 
selected. Accordingly, the pseudo-emotions can be indi- 
cated in a complex manner. 55 
[0051] The emotion expression decision unit D deter- 
mines emotion expression patterns from the pseudo- 
emotions at the moment in the above-described man- 
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ner, set the patterns as emotion expression behavior, 
and outputs them to the behavior planning unit G. 

Autonomous Behavior Generation Unit 

[0052] Figure 8 is a schematic flow chart showing 
processes at the autonomous behavior generation unit 
E. 

[0053] The autonomous behavior generation unit E 
weighs plural autonomous behaviors set in an autono- 
mous behavior pattern memory 7 based on specific 
external information, the user information, the pseudo- 
emotion information, and the pseudo-personality infor- 
mation, in order to determine a degree of priority of 
each autonomous behavior. 

[0054] In this embodiment, the autonomous behavior 
patterns are composed of upper behavior patterns and 
lower behavior patterns. First, the upper behavior pat- 
terns are weighed, and after selecting an upper behav- 
ior pattern having a higher degree of importance, the 
lower behavior patterns are weighed for the selected 
upper behavior pattern, thereby selecting a lower 
behavior pattern having a higher degree of importance. 
[0055] The upper behavior patterns include, for exam- 
ple, "indication of newly released information", "speak- 
ing to the user", and "automatic collection of 
information". 

[0056] Each of the plural behavior patterns is corre- 
lated with the levels of each of the external information, 
the user information, the pseudo-emotion information, 
and the pseudo-personality, and the degree of impor- 
tance of each behavior is determined based on the level 
of each item of information. In practice, for example, 
with regard to the external information and the user 
information, the status of updating newly released infor- 
mation, its amount, the user's emotions, preferences, 
and personality, intervals of response by the user, and a 
time period while the user is located within a range 
where the system can locate the user are converted to 
numerical values which are then correlated with each 
degree of importance. With regard to the pseudo-emo- 
tion information and the pseudo-personality information, 
the level of each preliminary emotion and the level of 
each preliminary personality are correlated with the 
degree of importance of each behavior pattern. Correla- 
tion of each item of information with each degree of 
importance can be achieved using functional equations 
if there is regularity between them, if there is no regular- 
ity between them, data indicating their relationship can 
be obtained through experiments in advance. 
[0057] The autonomous behavior generation unit E 
assigns a degree of importance to each behavior pat- 
tern as described above, and then sets a behavior 
which has the highest degree of importance and which 
has not been selected for a given period of time since 
the same behavior was previously conducted, as auton- 
omous behavior information. In the above, the set 
behavior pattern is an upper behavior pattern, and the 
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autonomous behavior generation unit E assigns 
degrees of importance to plural lower behavior patterns 
for the set upper behavior pattern. The lower behavior 
patterns include, for example, "joking", "questioning", 
"reporting the current status", and "growling", which cor- 
respond to "speaking to the user" of the upper behavior 
pattern. 

[0058] The autonomous behavior generation unit E 
assigns degrees of importance to the lower behavior 
patterns, and then sets a behavior which has the high- 
est degree of importance as autonomous behavior 
information. In an embodiment indicated in Figure 8, the 
upper behavior pattern "speaking to the user" is 
selected, and further "questioning" is selected, i.e., the 
behavior "questioning" is selected. From data regarding 
the lower behavior patterns, one particular question is 
then selected, and this behavior is set as a final autono- 
mous behavior. In the above, when one data is selected 
from a group of data ranking lower (lower level or more 
detailed level) than the lower behavior patterns, the 
selection can be made at random or can be made in 
accordance with appropriate rules, e.g., if the command 
from the user can be understood from the user informa- 
tion, words in the text of the command can be used as 
keywords for a keyword search, and a question in corre- 
spondence most with the searched keyword can be 
selected. 

[0059] In this embodiment, assigning degrees of 
importance to behavior patterns are conducted in two 
levels, i.e., in both the upper behavior patterns and the 
lower behavior patterns, and final autonomous behavior 
information is selected from a group of low-level data. 
The selection process Is not limited to the above, and 
can be conducted in three or more levels when behavior 
patterns are categorized into three or more levels. If the 
levels of autonomous behavior patterns, to which 
degrees of importance are assigned, are categorized in 
detail, the range of autonomous behavior can be broad- 
ened. 

[0060] Further, in this embodiment, automatic collec- 
tion of information is one of many autonomous behavior 
patterns. However, automatic collection of information 
can be independent from the other autonomous behav- 
ior patterns, and can be conducted constantly. In this 
case, intervals of collecting information and a range of 
information only may be changed in accordance with 
the pseudo-personality information, the pseudo-emo- 
tion information, and the user information. That is, the 
above information and the levels of the automatic col- 
lection of information may be correlated with each other 
in advance. For example, the higher the level of "aggres- 
siveness' in the pseudo-personality, or the higher the 
level of "joy" in the pseudo-emotions, or the higher the 
level of "impatient character" in the user information, the 
higher the level of the automatic collection of informa- 
tion becomes, i.e., accordingly, intervals of collection 
becomes shorter and the range of information become 
wider. In this way, while the user is; interacting with the 



system, the system can collect new information by its 
own judgment, and the thus-obtained data provide 
higher satisfaction to the user by generation of more 
complete autonomous behavior and commanded 
5 behavior described below. 

Commanded Behavior Generation Unit 

[0061] The commanded behavior generation unit F 

10 generates commanded behavior by using information 
from the external information memory 3 in accordance 
with the user's command included in the user's com- 
mand information. In practice, in this assist system, the 
commanded behavior includes answering the user's 

15 questions, outlining an action plan required by the user, 
constituting a detailed action plan, and formation of sim- 
ulation of the action plan. More specifically, answering 
the user's questions means conducting a search using 
the external information memory 3 and/or the external 

20 databases and displaying the search result, if the user 
commands the system to conduct a search. Outlining 
an action plan and constructing a detailed plan mean 
laying out a travel plan showing the user's destination 
and tourist attractions on the way to the destination, lay- 

25 ing out a route if the user is touring by car or motorbike, 
and laying out a train route if the user is travelling by 
train. 

[0062] In the above, commanded behavior generation 
processes at the commanded behavior generation unit 

30 F are conducted by at least referring to the previous 
behavior outputted from the system, and the information 
about the user's evaluation as to the previous behavior. 
For example, when the user commands the system to 
conduct a search, if the previously outputted behavior 

35 included a result of a search, the commanded behavior 
generation unit F determines whether the keyword used 
in the previous search was in accordance with the 
user's request based on the level of the user's evalua- 
tion as to the result of the previous search. If the level of 

40 the user's evaluation was high, the unit F judges that the 
user is satisfied with the keyword used in the search, 
and the unit F conducts a second search with the same 
keyword. If the level of the user's evaluation was low, the 
unit F judges that the user is not satisfied with the key- 

45 word, the unit F conducts a new first search with a new 
keyword. 

[0063] As described above, upon referring to the 
user's evaluation and the previously outputted behavior, 
the next commanded behavior is generated, so that 

so behavior more suitable to the user can be generated. 
[0064] Further, the commanded behavior generation 
unit F can generate absolute behavior in addition to the 
above-mentioned commanded behavior. In practice, for 
example, risk-related keyword, such as "earthquake", 

55 "frequently happening", and "heavy rain and flood warn- 
ing", or generally Undesirable keywords are memorized 
as risk information keywords, and if such risk informa- 
tion keywords are found in a piece of information auto- 
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matically collected during autonomous behavior, the 
piece of information can be designed to be displayed as 
absolute behavior. As described above, by autono- 
mously detecting risk information and setting the infor- 
mation as absolute behavior, the risk information can be 
given priority over other information and be displayed in 
addition to an action plan, when the user's action is trav- 
elling. 

Behavior Planning Unit 

[0065] The behavior planning unit G receives behavior 
information from each of the emotion expression deci- 
sion unit D, the autonomous behavior generation unit E, 
and the commanded behavior generation unit F; sets 
degrees of priority at each item of behavior information 
based on the pseudo-personality information, the 
pseudo-emotion information, and the user information; 
generates new priority behavior information; combines 
each item of behavior information based on the new pri- 
ority behavior information having priority; and decides 
final output behavior. 

[0066] Figures 9 and 1 0 are schematic flowcharts 
showing processes at the behavior planning unit G. 
[0067] The behavior planning unit G first judges 
whether or not there is required absolute behavior. At 
this time, if there is required absolute behavior, the 
absolute behavior information is set at priority 1 . The 
absolute behavior information includes information on 
absolute behavior generated at the commanded behav- 
ior generation unit, as described above, and information 
on behavior requiring urgent action against a situation 
such as input device failure. 

[0068] As the result of judgment as to whether there is 
required absolute behavior, if there is no required abso- 
lute behavior, a degree of priority (1 to 3: in this case, 
priority is 1, i.e., high priority) is determined for each 
item of behavior information based on a table of 
degrees of priority, and is set as new priority behavior 
information. 

[0069] The table of degrees of priority is a table defin- 
ing degrees of priority as described in Figure 9, for 
example: If the pseudo-emotion is "joy", the pseudo- 
personality is "obedience", and the state of the user is 
"anger", the commanded behavior is set at priority 1 , the 
emotion expression behavior is set at priority 2, and the 
autonomous behavior is set at priority 3. As above, the 
table is a group of data wherein a degree of priority of 
each item of behavior information is set for a combina- 
tion of all pseudo-emotions, all pseudo-personalities, 
and all user's states. However, the method of determin- 
ing the degree of priority is not limited to the above, and 
includes, for example, fuzzy logic technique using fuzzy 
rules which define in advance the relationship between 
the pseudo-emotions, the pseudo-personalities, the 
user's states, and the degrees of priority of behavior 
information, wherein membership equations of behavior 
information are formulated with respect to the degrees 



of priority. 

[0070] After setting the new priority behavior informa- 
tion, the behavior which has already been performed 
and the behavior which need not be performed are 

5 deleted, and the remaining behavior patterns are set as 
incomplete behavior information. This incomplete 
behavior information is set to be more important than 
the aforesaid new priority behavior information. Based 
on the incomplete behavior information and the new pri- 

w ority behavior information, the current priority behavior 
information is updated. Each item of behavior informa- 
tion is combined based on the current priority behavior 
information. 

[0071] Figure 10 is a schematic flow chart showing 

15 processes from generation of each item of behavior 
information based on the current behavior information 
to setting of final output behavior. 
[0072] Composite behavior is created by setting a 
behavior having a higher degree of priority first. That is, 

20 each item of behavior information is composed of plural 
behaviors, and in practice, the behavior information of 
this embodiment includes any of the following: eyes, 
mouth, conversation, display, search, and file. In order 
of priority, behaviors in the behavior information are set, 

25 and a behavior in the behavior information having lower 
priority is set only when there is no conflict with the 
behaviors already defined by the higher priority behav- 
ior information, i.e., the behaviors which the high priority 
behavior information does not define can be filled with 

30 those in the lower priority behavior information. 

[0073] In the embodiment indicated in Figure 10, the 
behavior (display: newly released information) defined 
by the commanded behavior which has the highest pri- 
ority is set first. Second, the behavior defined by the 

35 emotion expression behavior having priority 2 (eyes: 
smiling eyes; mouth: smiling mouth; conversation: whis- 
tling) is set. Third, the behavior defined by the com- 
manded behavior having priority 3 (search: hot spring) 
is set. Fourth (last), the behavior defined by the autono- 

40 mous behavior having priority 4 is set. As can be seen 
in the above example, although the autonomous behav- 
ior having priority 4 defines "conversation: chatting (type 
3)", the behavior has been defined by the emotion 
expression behavior having priority 2 (conversation: 

45 whistling), i.e., "conversation:" has already been 
defined. Thus, the behavior "conversation" defined by 
the autonomous behavior is not adopted. In the above, 
the reason that there are two commanded behaviors 
with regard to "conversation" in the above example is 

so that: "commanded behavior information" in the previous 
incomplete behavior was maintained, i.e., the previous 
incomplete behavior information is assigned priority. 

Effects 

55 

[0074] As described above, according to this assist 
system, the assist system itself is equipped with a 
pseudo-personality, which grows by interacting with the 
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state of the user, and pseudo-emotions, which change 
in accordance with the state of the user and the external 
information wherein the pseudo-emotions are controlled 
by the pseudo-personality, and the pseudo-personality 
is different depending on the user, i.e., the pseudo-per- 5 
sonality unique to each user is being established in 
accordance with the use of the system. Since the 
pseudo-personality is expressed as the pseudo-emo- 
tions, the user can enjoy fostering a unique assist sys- 
tem during a growth period, similar to raising a child, w 
Further, approximately after completion of the growth 
period, the user can enjoy establishing "friendship" with 
the assist system. 

[0075] Further, the system is constituted so as to gen- 
erate autonomous behavior based on the pseudo-per- 15 
sonality and the pseudo-emotions controlled by the 
pseudo-personality. Thus, the more the system is used, 
the more the understanding of the user's feelings or 
preferences can be achieved. The system can assist 
the user in matters which the user does not directly 20 
request. 

[0076] Further, since the assist system can perform 
both autonomous behavior and commanded behavior, 
the user need not give detailed commands from the 
beginning. The user can set conditions through dia- 25 
logue between the system and the user. In addition, dur- 
ing the dialogue, the assist system can give its own 
suggestions (such as newly released information). 
Thus, an appropriate action plan can be established 
very easily with the assist system. 30 

Pet Robot 

[0077] With reference to Figure 11-19, another 
embodiment to which the control system of the present 35 
invention is adopted will be explained. This embodiment 
is a pet robot which a user can keep as a pet. 
[0078] Incidentally, as can be seen in Figures 11-19, 
the control system of the pet robot is almost the same 
as that in the assist system, except that the details of 40 
information at each processing unit, output devices, and 
the details of output information are different. Thus, the 
principle of the control system will not be explained 
again to avoid repetition. The different aspects from the 
assist system will be explained below. 45 
[0079] The pet robot is designed to give a user para- 
experience of keeping a pet. The pet robot itself has a 
pseudo-personality, which grows by interacting with the 
user and through its own experience, and pseudo-emo- 
tions, which change in accordance with interaction with so 
the user and the surrounding environments. Depending 
on the state of the pseudo-personality and the pseudo- 
emotions, the pet robot expresses emotions and 
responds to the user with autonomous response and/or 
commanded response. As a result, the pet robot is 55 
capable of conducting various actions. 
[0080] In practice, in order to recognize the state of 
the user or the environments, this pet robot is equipped 



with a visual detection sensor, a touch detection sensor, 
a hearing-detection sensor, and a means for accessing 
external databases, and further equipped with an output 
device, such as a visual output device and an auditory 
output device, for indicating its pseudo-emotions or 
responses to the user, and further with travelling means 
functioning as its hands and feet, an operation device, 
and a tail. 

Basic Structures of Pet Robot 

[0081] Figure 1 1 is a schematic block diagram show- 
ing an example of the entire system of a pet robot 
according to the present invention. As shown in this fig- 
ure, the pet robot comprises: a user/environment infor- 
mation recognition unit A for recognizing a state of the 
user or environment information; a pseudo-personality 
information-setting unit B for setting a pseudo-personal- 
ity by referring to the outcome of recognition by the rec- 
ognition unit A; a pseudo-emotion generation unit C for 
generating pseudo-emotions based on the outcome of 
recognition by the recognition unit A and the information 
on the pseudo-personality set by the pseudo-personal- 
ity information-setting unit B; an emotion expression 
decision unit D for deciding manners of emotion expres- 
sion, such as expression methods to express laughing 
or angry on a display, based on the information on the 
pseudo-emotion; an autonomous behavior generation 
unit E for generating autonomous behavior based on 
the outcome of recognition, the information on the 
pseudo-personality, and the information on the pseudo- 
emotions; a commanded behavior generation unit F for 
generating behavior in accordance with the user's com- 
mand based on the outcome of recognition and at least 
the preceding behavior previously outputted by this sys- 
tem itself, and a behavior planning unit G for deciding 
final behavior to be outputted using output from each of 
the emotion expression decision unit D, the autono- 
mous behavior generation unit E, and the commanded 
behavior generated unit F, based on the outcome of rec- 
ognition, the information on the pseudo-personality, and 
the information on the pseudo-emotions. As shown in 
Figure 12, once the system is activated, this assist sys- 
tem repeats each processing step at units A through G 
until the system is turned off. The structure and function 
of each unit will be explained in detail below. 

User/Environment Information Recognition Unit 

[0082] As shown in Figure 11, the user/environment 
information recognition unit A comprises various input 
devices (e.g., an auditory input device, a visual input 
device, a touch detection device, and/or an environment 
information detection device), a user's evaluation recog- 
nition unit Al, a user's emotion recognition unit A2, a 
user recognition unit A3, a user's command recognition 
unit A4, an environment information recognition unit A5, 
and a current user information/characteristics deduction 
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unit A6. The user's evaluation recognition unit Al per- 
forms recognition-processing based on the information 
obtained via the 'Input devices. The current user infor- 
mation/characteristics deduction unit A6 determines or 
deduces a current state of the user and the user's char- 5 
acteristics such as the user's preferences based on the 
recognition outcome from the user-related recognition 
units Al through A3. 

[0083] In the above, the input device includes a micro- 
phone as an auditory input device, a camera as a visual 10 
input device, a tactile sensor as a touch detection 
device, and an illumination sensor' a voltage sensor, an 
obstacle-sensing sensor, and a drive-sensing sensor 
(sensing a distance driven by a drive device as an envi- 
ronment information detection device. However, the 15 
input devices are not limited to the above. Any input 
device can be used which is available on the market, 
and which is capable of detecting the user's state and 
environment information visually, auditorily, and/or tac- 
tilely. 20 
[0084] Figure 1 3 is a schematic block diagram show- 
ing the user/environment information recognition unit A 
in detail to explain processes at each of the processing 
units A1 through A6 in the user/environment information 
recognition unit A. 25 
[0085] As shown in this figure, the user's evaluation 
recognition unit A1 establishes the user's evaluation as 
to behavior outputted by the pet robot and sets the 
result, for example, as information on the user's evalua- 
tion, based on the manner in which the user responds to 30 
the pet robot. The manner may be sensed by a tactile 
sensor, e.g., how the user touches or strokes the pet 
robot. If the manners of touching or stroking are corre- 
lated with evaluation values in advance, the evaluation 
values can be determined. 35 
[0086] The user's emotion recognition unit A2 pre- 
sumes the user's current emotions based on the char- 
acteristics found in the auditory information and the 
user's facial expression, wherein an auditory analysis 
(analysis of frequency distributions or pitches of the 40 
sound/voice) and a facial expression analysis (analysis 
based on the shapes of the eyes and the mouth) are 
conducted based on the auditory information and the 
user's facial expression inputted from the auditory input 
device and the visual input device. Further, based on 45 
the presumed emotions, the user's emotion recognition 
unit A2 determines the user's current emotion. 
[0087] The user recognition unit A3 recognizes 
whether the person detected by the pet robot is the 
user, based on the outcome! of recognition of the posi- so 
tion of the person and the body-type of the person by 
using a visual input device, and based on the outcome 
of analysis of the sound/voice obtained at the user emo- 
tion recognition unit A2. 

[0088] The user's command recognition unit A4 rec- 55 
ognizes and establishes the user's command informa- 
tion based on command information obtained by 
analyzing the meaning of the sound/voice recognized 
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by the auditory input device and by analyzing informa- 
tion on the user's gestures, recognized by the visual 
input device. For example, if the user says "fetch" and 
"points out a ball with his/her gesture, based on a com- 
bination, the user's command information is set as fol- 
lows: The command is "fetching" and the object is "the 
ball". 

[0089] The environment information recognition unit 
AS recognizes the location of an object based on infor- 
mation obtained from the visual input device; recog- 
nizes the travelling route of the pet robot when travelling 
autonomously based on information obtained from the 
drive-sensing sensor of the drive device; converts infor- 
mation, such as brightness, the presence or absence of 
an obstacle, and the electrical power level of the pet 
robot, obtained from various detection sensors, to 
appropriate forms; sets the converted information; and, 
based thereon, sets information on the environments 
surrounding the pet robot and the internal state of the 
pet robot. 

[0090] The current user information/characteristics 
deduction unit A6 determines or deduces the current 
state and the characteristics of the user based on the 
recognition results at the user-related information rec- 
ognition units All through A4, the information on the cur- 
rent priority behavior described later, the information on 
the emotion expression patterns, and the information on 
the autonomous behavior patterns. The current user 
information/characteristics deduction unit A6 then 
updates data of the recognition information related to 
the user stored in a user information memory 1 . 
[0091] In the above, the current state of the user 
includes the level of the user's emotions, the user's eval- 
uation as to the behavior outputted by the system, and 
the location of the user. The characteristics of the user 
includes frequency of the occurrence of each emotion, 
the change rate or range of emotions, and the average 
value of user's evaluation as to the behavior outputted 
by the system. 

[0092] The output of the user's command recognition 
unit A4 is, transmitted to a user's command information 
memory 2, thereby updating the information related to 
the user's command stored in the memory 2. The infor- 
mation related to the user's command includes, for 
example, the contents of the command, the object of the 
command, and the conditions of the command. 
[0093] The output of the external information recogni- 
tion unit A5 is transmitted to an external information 
memory 3, thereby updating the information stored in 
the memory 3. 

Pseudo-Personality Information-Setting Unit 

[0094] Figure 14 is a schematic flow chart showing 
processes at the pseudo-personality information-setting 
unit B. 

[0095] The pseudo-personality information-setting 
unit B is constructed so as to set, as a pseudo-person- 
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ality, a group of several preliminary personalities, each 
subdividable into several levels. Based on the informa- 
tion stored in the user information memory 2, the level of 
each preliminary personality is set, and the higher the 
level, the more the preliminary personality contributes to 
the formation of the pseudo-emotions and the action 
plan. 

[0096] As can be seen, processes at the pseudo-per- 
sonality information-setting unit B are substantially the 
same as those in the assist system. The details of the 
processes are therefore omitted. 

Pseudo-Emotion Generation Unit 

[0097] Figure 1 5 is a schematic flow chart showing 
processes at the pseudo-emotion generation unit C. 
[0098] The pseudo-emotion generation unit C 
receives output from the user information memory 1 , the 
external information memory 3, and the pseudo-person- 
ality information memory 4, and, based on the above 
information, the pseudo-emotion generation unit C gen- 
erates pseudo-emotion information of the system. 
[0099] As can be seen, processes at the pseudo-emo- 
tion generation unit G are substantially the same as 
those in the assist system. The details of the processes 
are therefore omitted. 

Emotion Expression Decision Unit 

[0100] Figure 16 is a schematic flow chart showing 
processes at the emotion expression decision unit D. 
[0101] The emotion expression decision unit D judges 
the level of each preliminary emotion from the pseudo- 
emotion information obtained from the pseudo-emotion 
memory 5, and determines emotion expression to be 
outputted based on preliminary emotions whose levels 
are high or more than zero. 

[0102] The emotion expression is selected from plural 
patterns of emotion expression in relation to pseudo- 
emotions stored in advance in an emotion expression 
pattern memory 6. The emotion expression includes, in 
practice, changes in shape of parts such as eyes and a 
mouth if a "face" is created by a visual output device. 
Further, emotion expression can be indicated by move- 
ment of a tail or the pet robot itself. In the above, the 
emotion expression pattern memory 6 memorizes in 
advance patterns of each of parts of "face", and pat- 
terns of movement of the tail or the pet robot itself in 
relation to each level of each preliminary emotion. 
[0103] The method of selecting the patterns of the 
emotion expressions at the emotion expression unit D 
can be simply a method wherein the pattern of the emo- 
tion expression, which corresponds to the preliminary 
emotion having the highest level, is selected. However, 
preferably, a pattern of the absence of expression 
(expressionless) is created among patterns of emotion 
expressions, and if the levels of all of the preliminary 
emotions are zero, the expressionless pattern is 



selected, and if the levels of preliminary emotions, 
except for one preliminary emotion, are zero, a pattern 
corresponding to the one preliminary emotion is 
selected. If the levels of more than one but not ail pre- 

5 liminary emotions are more than zero, a pattern of a 
preliminary emotion having a higher level than those of 
the other preliminary emotions has priority over the 
other patterns. Further, among the remaining patterns, 
a pattern of a preliminary emotion having a lower level 

10 than the first one is selected where the first pattern does 
not apply. In the. above, the first pattern and the second 
pattern are combined to make one combined expres- 
sion. If the levels of all of the preliminary emotions are 
more than zero, the expressionless pattern may be 

15 selected. Accordingly, the pseudo-emotions can be indi- 
cated in a complex manner. 

[0104] The emotion expression decision unit D deter- 
mines emotion expression patterns from the pseudo- 
emotions at the moment in the above-described man- 
20 ner, sets the patterns as emotion expression behavior, 
and outputs them to the behavior planning unit G. 

Autonomous Behavior Generation Unit 

25 [0105] Figure 17 is a schematic flow chart showing 
processes at the autonomous behavior generation unit 
E. 

[0106] The autonomous behavior generation unit E 
determines each action level of plural itemized autono- 

30 mous behaviors set in an autonomous behavior pattern 
memory 7 based on each level of the environment infor- 
mation, the user information, the pseudo-emotion infor- 
mation, and the pseudo-personality information, in 
order to determine a degree of priority of each itemized 

35 autonomous behavior. In the above, each of the envi- 
ronment information, the user information, the pseudo- 
emotion information, and the pseudo-personality infor- 
mation is subdivided into plural levels. 
[0107] In this embodiment, the itemized autonomous 

40 behavior patterns are prepared for all of changeable or 
movable parts (such as the eyes, the mouth, voices, the 
height of the end of the tail, the movement of the tail, 
and the travelling of the pet robot itself). Each itemized 
autonomous behavior is correlated with the levels of 

45 each item of information. By adding or subtracting 
action levels of each itemized autonomous behavior 
based on the level of each item of information, the 
action level of each itemized autonomous behavior is 
determined. 

50 

Commanded Behavior Generation Unit 

[0108] The commanded behavior generation unit F 
generates commanded behavior by using information 
55 from the environment information memory 3 in accord- 
ance with the user's command included in the users 
command information. In practice, in this embodiment, if 
the users commanded information is (command: fetch- 
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ing; object: bail), the unit F generates behavior informa- 
tion including recognition of the location of the ball 
based on information from the environment information 
memory, moving to the location, fetching the ball, and 
returning to the user. 

Behavior Planning Unit 

[0109] The behavior planning unit G receives; behav- 
ior information from each of the emotion expression 
decision unit D, the autonomous behavior generation 
unit E, and the commanded behavior generation unit F; 
sets degrees of priority at each item of behavior infor- 
mation based on the pseudo-personality information, 
the pseudo-emotion information, and the user informa- 
tion; generates new priority behavior information; com- 
bines each item of behavior information based on the 
new priority behavior information having priority; and 
decides final output behavior. 

[0110] Figures 18 and 19 are schematic flow charts 
showing processes at the behavior planning unit G. 
[0111] The behavior planning unit G first judges 
whether or not there is required absolute behavior. At 
this time, if there is required absolute behavior, the 
absolute behavior information is set at priority 1 . The 
absolute behavior information includes information on 
behavior requiring urgent action against a situation such 
as battery power shortage and arrest by collision. The 
conditions corresponding to the absolute behavior are 
memorized in the behavior generation unit G. 
[01 1 2] As the result of judgment as to whether there is 
required absolute behavior, if there is no required abso- 
lute behavior, a degree of priority (1 to 3: in- this case, 
priority is 1, i.e., high priority) is determined for each 
item of behavior information based on a table of 
degrees of priority, and is set as new priority behavior 
information. 

[01 1 3] The table of degrees of priority is a table defin- 
ing degrees of priority as described in Figure 18, for 
example: lithe pseudo-emotion is "joy", the pseudo-per- 
sonality is "obedience", and the state of the user is 
"anger", the commanded behavior is set at priority 1 , the 
emotion expression behavior is set at priority 2, and the 
autonomous behavior is set at priority 3. As above, the 
table is a group of data wherein a degree of priority of 
each item of behavior information is set for a combina- 
tion of all pseudo-emotions, all pseudo-personalities, 
and all user's states. However, the method of determin- 
ing the degree of priority is not limited to the above, and 
includes, for example, fuzzy logic technique using fuzzy 
rules which define in advance the relationship between 
the pseudo-emotions, the pseudo-personalities, the 
user's states, and the degrees of priority of behavior 
information, wherein membership equations of behavior 
information are formulated with respect to the degrees 
of priority. 

[01 1 4] After setting the new priority behavior informa- 
tion, the behavior which has already been performed 



and the behavior which need not be performed are 
deleted, and the remaining behavior patterns are set as 
incomplete behavior information. This incomplete 
behavior information is set to be more important than 

5 the aforesaid new priority behavior information. Based 
on the incomplete behavior information and the new pri- 
ority behavior information, the current priority behavior 
information is updated. Each item of behavior informa- 
tion is combined based on the current priority behavior 

w information. 

[0115] Figure 19 is a schematic flow chart showing 
processes from generation of each item of behavior 
information based on the current behavior information 
to setting of final output behavior. 

is [0116] Composite behavior is created by setting a 
behavior having a higher degree of priority first. That is, 
each item of behavior information is composed of plural 
behaviors, and in this embodiment indicated in Figure 
19, the behavior information of this embodiment 

20 includes any of the following: eyes, mouth, conversa- 
tion, travelling, and information. In the order of priority, 
behaviors in the behavior information are set, and a 
behavior in the behavior information having lower prior- 
ity is set only when there is no conflict with the behav- 

25 iors already defined by the higher priority behavior 
information, i.e., the behaviors which the high priority 
behavior information does not define can be filled with 
those in the lower priority behavior information. 
[0117] In the embodiment indicated in Figure 19, the 

30 behavior (travelling: approaching the user) defined by 
the commanded behavior which has the highest priority 
is set first. Second, the behavior defined by the emotion 
expression behavior having priority 2 (eyes: smiling 
eyes; mouth: smiling mouth; conversation: whistling; 

35 travelling: swaying the body) is set. Third, the behavior 
defined by the commanded behavior having priority 3 
(Information: indicating the remaining battery power 
level) is set. Fourth (last), the behavior defined by the 
autonomous behavior having priority 4 (conversation: 

40 whistling; travelling: moving to a brighter place) is set. 
As can be seen in the above example, although the 
autonomous behaviors having priorities 1 , 2, and 4 have 
information on "travelling", only the information on "trav- 
elling" in the behavior having priority 1 is employed, and 

45 the information on "travelling" in the behavior having 
lower priorities is not adopted. In the above, the reason 
that there are two commanded behaviors with regard to 
"travelling" in the above example is that "commanded 
behavior information" in the previous incomplete behav- 

so ior was maintained, i.e., the previous incomplete behav- 
ior information is assigned priority. 

Effects 

55 [01 18] As described above, according to this system, 
the pet robot itself is equipped with a pseudo-personal- 
ity, which grows by interacting with the state of the user, 
and pseudo-emotions, which change in accordance 
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with the state of the user and the environment informa- 
tion, wherein the pseudo-emotions are controlled by the 
pseudo-personality, and the pseudo-personality is dif- 
ferent depending on the user, i.e. the pseudo-personal- 
ity unique to each user is being established in 
accordance with the use of the pet robot. Since the 
pseudo-personality is expressed as pseudo-emotions, 
the user can enjoy fostering a unique pet robot during a 
growth period, similar to raising a child. 'Further, 
approximately after completion of the growth period, the 
user can enjoy establishing "friendship" with the pet 
robot. 

[0119] Further, the pet robot is constituted so as to 
generate autonomous behavior based on the pseudo- 
personality and the pseudo-emotions controlled by the 
pseudo-personality. Thus, the more the pet robot is 
used, the more the personality of the pet robot is estab- 
lished, thereby allowing the pet robot to behave in vari- 
ous manners. 

[0120] Further, since the pet robot can perform both 
autonomous behavior and commanded behavior, the 
pet robot will never be completely delinquent nor com- 
pletely obedient. Various actions of the pet robot keep 
the user interest in the pet robot. 

Technical Options 

[0121] A robotics platform suitable for mobile naviga- 
tion tasks can be designed to provide on-board 
audio/video capabilities and a radio link to an off-board 
computer. Existing techniques for color tracking to track 
the user's hands and head location can be adapted. 
Existing word spotting speech recognition software for 
recognizing simple one- or two-word spoken commands 
can also be adapted. Any methods for simple non-ver- 
bal audio analysis, such as verbal intonation analysis to 
recover the affective or emotion content of a command 
can be adapted. Existing schema-based reactive navi- 
gation methods can be adapted to provide the system 
with the capability of navigating around its environment 
without colliding with obstacles. In the above, explora- 
tion behaviors as well as goal seeking behaviors for 
moving to specific destinations can be encoded. Fur- 
ther, "reasoning" capabilities and a repertoire of behav- 
iors can be formed with an appropriate decision-making 
algorithm that takes into account the system's goals and 
goal priorities, the current environment or situation, and 
user's goals and emotion state, for example. Finally, any 
algorithms can be adapted for adapting the system's 
behavior in response to unexpected situations, for 
learning new behaviors, for learning the environment, 
and for learning about the user's preferences. The func- 
tioning of the system can be organized by, for example, 
expanding a behavior-based subsystem (responsible 
for low level interaction of the system with the environ- 
ment) with high-level processing modules (responsible 
for "high-level" interaction of the system with the user) 
that work in parallel. 



[01 22] Further, correlations between various informa- 
tion and various variables can be determined in 
advance using existing techniques such as neural net- 
works, fuzzy neural networks, and genetic algorithms if 

5 the correlations are highly complex, or using existing 
techniques such as maps and functional equations if the 
correlations are rather simple. In this regard, Da Ruan 
(editor)" Intelligent Hybrid Systems - Fuzzy Logic, Neu- 
ral Networks, and Genetic Algorithms 2' Kluwer Aca- 

10 demic Publishers (1997), J 1.-5. R. Jang, C.J. Sun, E. 
Mizutani, "Neuro-Fuzzy and Soft Computing" Prentice 
Hall Upper Saddle River, NJ 07458 (1997), C.-T. Lin and 
C. 5. George Lee, "Neural Fuzzy Systems" Prentice Hall 
Upper Saddle River, NJ 07458 (1998), and N.K. Kasa- 

15 bov, "Foundations of Neural Networks, Fuzzy Systems, 
and Knowledge Engineering" the MIT Press (1996) are 
hereby incorporated by reference. The above tech- 
niques can be combined, and learning control can be 
adapted for any techniques. 

20 

Other Applications 

[0123] The object to be controlled in the present 
invention is not limited, and any objects, as in the first 

25 and second embodiments, such as an engine in a 
motorbike or a car, an outboard engine in a vessel, an 
industrial robot used in machine tools, a motor used in 
electrically-driven vehicles, or the like can be controlled 
by adopting the control system of the present invention 

30 based on the same principle as in the aforesaid embod- 
iments. Incidentally, the external information or environ- 
ment information can be omitted if the object is used 
under limited circumstances or if the external or envi- 
ronment information has nothing to do with accomplish- 

35 ment of the user's command or if the user so desires. 
[0124] When the present invention is adapted to a 
control object in a different field or for a different pur- 
pose, by modifying items included in information to be 
processed, the control object can be controlled as with 

40 the aforesaid embodiments. In any case, according to 
the present invention, the control object, to which the 
present invention is adapted, is likely to be equipped 
with quasi-intelligence. 

[0125] It will be understood by those of skill in the art 
45 that numerous variations and modifications can be 
made without departing from the spirit of the present 
invention. Therefore, it should be clearly understood 
that the forms of the present invention are illustrative 
only and are not intended to limit the scope of the 
50 present invention. 

Claims 

1. An autonomous device behaving adaptively to a 
55 user, comprising: 

a recognition unit which recognizes a user- 
related status; 
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a personality-forming unit which establishes 
personality composed of preselected personal- 
ity elements based on the user-related status 
accumulated for a predetermined period of 
time based on a predetermined relationship 
between the selected personality elements and 
the user-related status; 

an emotion-forming unit which establishes 
emotions composed of preselected emotion 
elements based on the user-related status and 
the personality based on a predetermined rela- 
tionship between the preselected emotion ele- 
ments, the user-related status, and the 
personality; and 

an autonomous behavior-establishing unit 
which selects a pattern of autonomous behav- 
ior based on the user-related status, the per- 
sonality, and the emotions, based on a 
predetermined relationship between patterns 
of autonomous behavior, the user-related sta- 
tus, the personality, and the emotions. 

2. The autonomous device according to claim 1, 
wherein the recognition unit further recognizes the 
user's command which is not included in the user- 
related status, the device further comprises: 

a commanded behavior-establishing unit which 
selects a pattern of commanded behavior in 
accordance with the recognized user's com- 
mand; and 

a behavior control unit which controls behavior 
by combining the selected pattern of autono- 
mous behavior and the selected pattern of 
commanded behavior based on a predeter- 
mined priority order. 

3. An autonomous device according to claim 1 or 2, 
further comprising: 

a sensing unit which senses signals from the 
user; 

a first memory which stores and updates the 
recognized user-related status and/or the rec- 
ognized command; and 
a second memory which stores and updates 
the formed personality, wherein updating takes 
place under predetermined conditions. 

4. The autonomous device according to claims 1 , 2 or 
3, wherein: 

the sensing unit further senses predetermined 
user-unrelated signals relevant to accomplish- 
ment of the command but not part of the com- 
mand; 

the recognition unit further recognizes a cur- 
rent user-unrelated status based on the sensed 



predetermined signals; 

the first memory further stores and instantane- 
ously updates the recognized user-unrelated 
status; 

5 the emotion-forming unit establishes each ele- 

ment of predetermined emotion elements 
based further on the user-unrelated status in 
addition to the user-related status stored in the 
first memory and the personality stored in the 

w second memory to form emotions; 

the predetermined relationship memorized in 
the emotion-forming unit further includes the 
user-unrelated status to define a relationship 
between the predetermined emotion elements, 

15 the user-related status, the user-unrelated sta- 

tus, and the personality; 
the autonomous behavior-establishing unit 
selects a pattern of autonomous behavior 
based further on the user-unrelated status in 

20 addition to the user-related status, the person- 

ality, and the emotions; and 
the predetermined relationship memorized in 
the autonomous behavior-forming unit further 
includes the user-unrelated status to define a 

25 relationship between patterns of autonomous 

behavior, the user-related status, the user- 
unrelated status, the personality, and the emo- 
tions. 

30 5. The autonomous device according to claim 3 or 4, 
further comprising a third memory which stores and 
updates the emotion formed by the emotion-form- 
ing unit, said third memory being linked to the 
behavior control unit, wherein the second memory 

35 is further linked to the behavior control unit, wherein 
the emotion stored in the third memory and the per- 
sonality stored in the second memory are used as 
parameters of the predetermined priority order pro- 
vided in the behavior control unit. 

40 

6. The autonomous device according to claim 5, 
wherein the first memory is further linked to the 
behavior control unit, wherein the user-related sta- 
tus stored in the first memory, the personality stored 
45 in the second memory, and the emotion stored in 
the third memory are used as parameters of the 
predetermined priority order provided in the behav- 
ior control unit. 

so 7. A method for adapting behavior of a device to a 
user's characteristics, comprising the steps of: 

(b) recognizing a user-related status; 
(d) establishing personality composed of 
55 preselected personality elements based on the 

user-related status accumulated for a predeter- 
mined period of time based on a predeter- 
mined relationship between the preselected 
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personality elements and the user-related sta- 
tus; 

(f) establishing emotions composed of prese- 
lected emotion elements based on the user- 
related status and the personality based on a s 
predetermined relation between the prese- 
lected emotion elements, the user-related sta- 
tus, and the personality; and 

(g) selecting a pattern of autonomous behavior 
based on the user-related status, the personal- 10 
ity, and the emotions based on a predeter- 
mined relationship between patterns of 
autonomous behavior, the user-related status, 

the personality, and the emotions. 

15 

8. The method according to claim 7, further compris- 
ing the steps of: recognizing the user's command 
which is not included in the user-related status; 

(h) selecting a pattern of commanded behavior 20 
in accordance with the recognized user's com- 
mand; 

(i) controlling behavior by combining the 
selected pattern of autonomous behavior and 

the selected pattern of commanded behavior 25 
based on a predetermined priority order. 



related status, the user-unrelated status, and 
the personality; 

in step (g), selecting a pattern of autonomous 
behavior based further on the user-unrelated 
status in addition to the user-related status, the 
personality, and the emotions, wherein the pre- 
determined relationship stored in the memory 
further includes the user-unrelated status to 
define a relationship between patterns of 
autonomous behavior, the user-related status, 
the user-unrelated status, the personality, and 
the emotions. 

11. The method according to claim 9 or 10, further 
comprising the steps of storing and updating the 
emotion formed in step (f), wherein the stored and 
updated emotion and the personality stored in step 
(e) are used as parameters of the predetermined 
priority order provided in step (i). 

12. The method according to claim 11, wherein the 
user-related status stored in step (c), the personal- 
ity stored in step (e), and the stored and updated 
emotion are used as parameters of the predeter- 
mined priority order in step (i). 



9. Method according to claim 7 or 8, further compris- 
ing the steps of: 

30 

(a) sensing signals from the user: 
(c) storing and updating the recognized user- 
related status and/or the recognized command; 
and 

(e) storing and updating the formed personality, 35 
wherein updating takes place under predeter- 
mined conditions. 



10. The method according to claim 9, further compris- 
ing: 40 

in step (a), sensing predetermined user-unre- 
lated signals relevant to accomplishment of the 
command but not part of the command; 
in step (b), recognizing a current user-unre- 45 
lated status based on the sensed predeter- 
mined signals; 

in step (c), storing and updating the recognized 
user-unrelated status; 

in step (d), establishing each element of prede- so 
termined emotion elements based further on 
the user-unrelated status in addition to the 
user-related status stored in step (c) and the 
personality stored in step (e) to form emotions, 
wherein the predetermined relationship stored 55 
in the memory further includes the user-unre- 
lated status to define a relationship between 
the predetermined emotion elements, the user- 
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